Method and apparatus for performing power analytics of a storage system

ABSTRACT

A storage system comprises one or more storage devices, power supplies supplying power to the storage device, a processor that performs in response to determining that the total power consumption of the one or more storage devices is less than a first percentage threshold of a load of the active power supplies, deactivating one or more of the active power supplies until the total power consumption is equal to or greater than the first percentage threshold of a load of each of the active power supplies, and in response to determining that the total power consumption is equal to or greater than a second percentage threshold of a load of each of the active power supplies, activating one or more of the deactivated ones of the power supplies until the total power consumption is less than the second percentage threshold of the load of each of the active power supplies.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 62/713,466, filed Aug. 1, 2018, the content ofwhich is hereby incorporated by reference in its entirety.

This application is also a continuation-in-part of U.S. patentapplication Ser. No. 15/975,463, filed May 9, 2018, entitled “METHOD ANDAPPARATUS FOR SELF-REGULATING POWER USAGE AND POWER CONSUMPTION INETHERNET SSD STORAGE SYSTEMS”, which claims priority to and the benefitof U.S. Provisional Application No. 62/638,035, filed Mar. 2, 2018, theentire contents of both of which are incorporated herein by reference.

BACKGROUND

Many companies provide cloud-based storage to end users so that endusers will have the ability to remotely access their stored data. Suchcompanies generally take advantage of Ethernet-attached solid statedrives (eSSDs) for their storage requirements. In particular,Ethernet-attached non-volatile memory express NVMe (Non-Volatile MemoryExpress) SSDs (e.g., NVMe Over Fabrics [NVMe-oF] storage devices) areconsidered an emerging and disruptive technology in this area.

Cloud-based storage providers typically charge users for storing theirdata on a monthly or annual basis based on the total storage spaceallocated to the user and either the average cost of energy consumed byall users or the maximum power consumption capable of being consumed bythe user based on the system. For example, for two users who havepurchased the same amount of cloud storage space, a user who stores onlya small amount of data relative to the total purchased storage space andonly stores data on an infrequent basis will be charged the same as auser who is regularly removing and added new data and using the majorityof his/her purchased storage space. Ideally, users should be charged forstorage based on the energy resources actually consumed. However, thereis no accurate method for calculating the power consumption ofindividual users, or calculating power consumption in real time.

The above information disclosed in this Background section is only forenhancement of understanding of the background of the disclosure andtherefore it may contain information that does not constitute prior art.

SUMMARY

Aspects of embodiments of the present invention are directed to astorage system, and a method of operating the same, capable of managing(e.g., optimizing) operation of the power supplies of the system bydynamically monitoring their operation and ensuring that active powersupplies operate in their high power-efficiency range.

Aspects of embodiments of the present invention are directed to astorage system, and a method of operating the same, capable of managing(e.g., optimizing) power usage of storage devices of a storage bank bydynamically adjusting their maximum power caps based on the workload ofthe storage bank.

According to some embodiments of the present invention, there isprovided a storage system comprising: one or more storage devices; aplurality of power supplies configured to supply power to the storagedevice; a processor; and a memory having stored thereon instructionsthat, when executed by the processor, cause the processor to perform:determining whether multiple power supplies of the plurality of powersupplies are active; in response to determining that multiple powersupplies are active: determining a total power consumption of the one ormore storage devices; in response to determining that the total powerconsumption is less than a first percentage threshold of a load ofactive ones of the power supplies, deactivating the active ones of thepower supplies one by one until the total power consumption is equal toor greater than the first percentage threshold of a load of each of theactive ones of the power supplies; and in response to determining thatthe total power consumption is equal to or greater than a secondpercentage threshold of a load of each of the active ones of the powersupplies, activating deactivated ones of the power supplies one by oneuntil the total power consumption is less than the second percentagethreshold of the load of each of the active ones of the power supplies.

In some embodiments, the determining the total power consumption of theone or more storage devices comprises: obtaining an actual powerconsumption of each storage device of the one or more storage devicesfrom the storage device or a corresponding power meter; and summing theactual power consumption of each storage device to obtain the totalpower consumption.

In some embodiments, the obtaining the actual power consumption of eachstorage device comprises: retrieving power measurement information froma power log corresponding to the storage device, wherein the powermeasurement information is measured, and recorded in the power log, bythe corresponding power meter.

In some embodiments, the corresponding power meter is internal to thestorage device.

In some embodiments, the corresponding power meter is external to andcoupled to the storage device.

In some embodiments, the first percentage threshold of the load of eachof the active ones of the power supplies is 40% of the load of each ofthe active ones of the power supplies.

In some embodiments, the second percentage threshold of the load of eachof the active ones of the power supplies is 90% of the load of each ofthe active ones of the power supplies.

In some embodiments, the instructions further cause the processor toperform: determining whether only one power supply of the plurality ofpower supplies is in a high availability mode; and in response todetermining that only one power supply of the plurality of powersupplies is in a high-availability mode, generating a warning messageindicating that the one power supply is in high-availability mode.

In some embodiments, the deactivating the active ones of the powersupplies one by one comprises: deactivating an active power supply ofthe active ones of the power supplies; determining that the total powerconsumption of the one or more storage devices is less than the firstpercentage threshold of a load of the active ones of the power supplies;and in response to the determining, deactivating an other active powersupply of the active ones of the power supplies.

In some embodiments, the activating the deactivated ones of the powersupplies one by one comprises: activating a deactivated power supply ofthe power supplies; determining that the total power consumption of theone or more storage devices is equal to or greater than the secondpercentage threshold of a load of the active ones of the power supplies;and in response to the determining, enabling an other deactivated powersupply of the power supplies.

According to some embodiments of the present invention, there isprovided a method of managing a storage system comprising one or morestorage devices and a plurality of power supplies configured to supplypower to the storage device, the method comprising: determining, by aprocessor of the storage device, whether multiple power supplies of theplurality of power supplies are active; in response to determining thatmultiple power supplies are active: determining, by the processor, atotal power consumption of the one or more storage devices; in responseto determining that the total power consumption is less than a firstpercentage threshold of a load of active ones of the power supplies,deactivating, by the processor, the active ones of the power suppliesone by one until the total power consumption is equal to or greater thanthe first percentage threshold of a load of each of the active ones ofthe power supplies; and in response to determining that the total powerconsumption is equal to or greater than a second percentage threshold ofa load of each of the active ones of the power supplies, activating, bythe processor, deactivated ones of the power supplies one by one untilthe total power consumption is less than the second percentage thresholdof the load of each of the active ones of the power supplies.

According to some embodiments of the present invention, there isprovided a storage system comprising: a plurality of storage devices,each storage device of the plurality of storage devices being configuredto measure a power consumption of the storage device; a processor incommunication with the plurality of storage devices; and a memory havingstored thereon instructions that, when executed by the processor, causethe processor to perform: determining whether one or more first storagedevices of the plurality of storage devices are idle or are in an idlestate; in response to determining that the one or more first storagedevices are in an idle state, instructing the one or more first storagedevices to operate at lower power caps; determining whether one or moresecond storage devices of the plurality of storage devices are consumingpower under a threshold power level; and in response to determining thatthe one or more second storage devices are consuming power under thethreshold power level, instructing the one or more second storagedevices to operate at or below the threshold power level.

In some embodiments, the determining whether one or more first storagedevices are in idle state: obtaining power consumption of each storagedevice of the plurality of storage devices by retrieving a correspondingpower log from the storage device; comparing the power consumption ofeach storage device with an idle power level; and determining whetherthe one or more first storage devices have power consumptions that areat or below the idle power level.

In some embodiments, the power log stores actual power consumption ofthe corresponding storage device as measured by a corresponding powermeter.

In some embodiments, instructing the one or more first storage devicesto operate at the lower power caps comprises: instructing the one ormore first storage devices to change power states to a power statehaving a lower maximum power rating.

In some embodiments, determining whether the one or more second storagedevices of the plurality of storage devices are consuming power under athreshold power level comprises: obtaining power consumption of eachstorage device of the plurality of storage devices by retrieving acorresponding power log from the storage device; comparing the powerconsumption of each storage device with the threshold power level; anddetermining whether the one or more first storage devices have powerconsumptions that below the threshold power level.

In some embodiments, instructing the one or more second storage devicesto operate at or below the threshold power level comprises: instructingthe one or more second storage devices to change power states to a powerstate having a maximum power rating corresponding to the threshold powerlevel.

In some embodiments, the instructions further cause the processor toperform: determining whether one or more storage slots are not occupiedby any storage device;

and in response to determining that the one or more storage slots arenot occupied by any storage device: identifying one or more power metersassociated with the one or more storage slots; and instructing theidentified one or more power meters to operate at lower power cap.

In some embodiments, instructing the identified one or more power metersto operate at lower power cap comprises: instructing the one or morepower meters to operate at a lowest power state.

In some embodiments, instructing the identified one or more power metersto operate at lower power cap comprises: instructing the one or morepower meters to deactivate.

BRIEF DESCRIPTION OF THE DRAWINGS

Further features and aspects will become apparent and will be bestunderstood by reference to the following detailed description reviewedin conjunction with the drawings. In the drawings:

FIG. 1 is an internal block diagram of a storage device according to anembodiment of the present invention.

FIG. 2 is a flow chart of a method for collecting power consumptionmeasurements from a power measurement unit in the storage device of FIG.1.

FIG. 3 is a schematic diagram of a storage system incorporating multiplestorage devices that are capable of providing power measurements.

FIG. 4 is a block diagram of an embodiment of the storage system of FIG.3 in which a PCIe switch is used.

FIG. 5 is a diagram depicting an embodiment in which power measurementsare transferred to a local service processor based on a query from thelocal service processor.

FIG. 6 is a diagram depicting an embodiment in which power measurementsare set by the local service processor.

FIG. 7 shows an example of a power policy which can be used to by thelocal service processor 50 to control power consumption of a storagedevice.

FIG. 8 is a diagram depicting an embodiment in which power measurementsare stored in a controller memory buffer until fetched by the localservice processor.

FIG. 9 is a diagram depicting an embodiment in which power measurementstaken by a power measurement unit are directly accessible to the localservice processor.

FIG. 10 is an example of a power log according to an embodiment of thepresent invention.

FIG. 11 is an illustrative method of how a storage system manages thepower reporting of multiple storage devices in its chassis using thepower log of FIG. 10.

FIG. 12 is a block diagram illustrating a storage system utilizing astorage bank and a power distribution unit, according to some exemplaryembodiments of the present invention.

FIGS. 13A-13D illustrate histograms of power consumption of a storagesystem as generated by the local service processor, according to someexemplary embodiments of the present invention.

FIG. 14 is flow diagram illustrating a process of managing the operationof the power supplies of a storage system, according to some exemplaryembodiments of the present invention.

FIG. 15 is flow diagram illustrating a process of managing the storagedevices of the storage system, according to some exemplary embodimentsof the present invention.

DETAILED DESCRIPTION

Hereinafter, example embodiments will be described in more detail withreference to the accompanying drawings, in which like reference numbersrefer to like elements throughout. The present invention, however, maybe embodied in various different forms, and should not be construed asbeing limited to only the illustrated embodiments herein. Rather, theseembodiments are provided as examples so that this disclosure will bethorough and complete, and will fully convey the aspects and features ofthe present invention to those skilled in the art. Accordingly,processes, elements, and techniques that are not necessary to thosehaving ordinary skill in the art for a complete understanding of theaspects and features of the present invention may not be described.Unless otherwise noted, like reference numerals denote like elementsthroughout the attached drawings and the written description, and thus,descriptions thereof will not be repeated. In the drawings, the relativesizes of elements, layers, and regions may be exaggerated for clarity.

It will be understood that when an element or layer is referred to asbeing “on,” “connected to,” or “coupled to” another element or layer, itcan be directly on, connected to, or coupled to the other element orlayer, or one or more intervening elements or layers may be present. Inaddition, it will also be understood that when an element or layer isreferred to as being “between” two elements or layers, it can be theonly element or layer between the two elements or layers, or one or moreintervening elements or layers may also be present.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the presentinvention. As used herein, the singular forms “a” and “an” are intendedto include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises,” “comprising,” “includes,” and “including,” when used inthis specification, specify the presence of the stated features,integers, steps, operations, elements, and/or components, but do notpreclude the presence or addition of one or more other features,integers, steps, operations, elements, components, and/or groupsthereof. As used herein, the term “and/or” includes any and allcombinations of one or more of the associated listed items. Expressionssuch as “at least one of,” when preceding a list of elements, modify theentire list of elements and do not modify the individual elements of thelist.

As used herein, the term “substantially,” “about,” and similar terms areused as terms of approximation and not as terms of degree, and areintended to account for the inherent deviations in measured orcalculated values that would be recognized by those of ordinary skill inthe art. Further, the use of “may” when describing embodiments of thepresent invention refers to “one or more embodiments of the presentinvention.” As used herein, the terms “use,” “using,” and “used” may beconsidered synonymous with the terms “utilize,” “utilizing,” and“utilized,” respectively. Also, the term “exemplary” is intended torefer to an example or illustration.

The electronic or electric devices and/or any other relevant devices orcomponents according to embodiments of the present invention describedherein may be implemented utilizing any suitable hardware, firmware(e.g. an application-specific integrated circuit), software, or acombination of software, firmware, and hardware. For example, thevarious components of these devices may be formed on one integratedcircuit (IC) chip or on separate IC chips. Further, the variouscomponents of these devices may be implemented on a flexible printedcircuit film, a tape carrier package

(TCP), a printed circuit board (PCB), or formed on one substrate.Further, the various components of these devices may be a process orthread, running on one or more processors, in one or more computingdevices, executing computer program instructions and interacting withother system components for performing the various functionalitiesdescribed herein. The computer program instructions are stored in amemory which may be implemented in a computing device using a standardmemory device, such as, for example, a random access memory (RAM). Thecomputer program instructions may also be stored in other non-transitorycomputer readable media such as, for example, a CD-ROM, flash drive, orthe like. Also, a person of skill in the art should recognize that thefunctionality of various computing devices may be combined or integratedinto a single computing device, or the functionality of a particularcomputing device may be distributed across one or more other computingdevices without departing from the spirit and scope of the exemplaryembodiments of the present invention.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which the present invention belongs. Itwill be further understood that terms, such as those defined in commonlyused dictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art and/orthe present specification, and should not be interpreted in an idealizedor overly formal sense, unless expressly so defined herein.

Embodiments of the present invention include a storage device, such asan SSD (e.g., NVMe or NVMe-oF SSD), that is capable of reporting itsactual power consumption to the local service processor, for example, abaseboard management controller (BMC). This enables the local serviceprocessor to provide power profiles and consumption of the storagedevice. In some embodiments, the storage device can report to the localservice processor or BMC via a system management bus (SMBus) or aPeripheral Component Interconnect Express (PCIe), and can report by oneof various protocols, such as by a Management Component TransportProtocol (MCTP) or by a NVMe Management Interface protocol for NVMe SSDsstorage devices. In some embodiments, the storage system may be anNVMe-oF based system. Further embodiments include a storage systemincluding several storage devices in which each storage device iscapable of reporting its actual power consumption to the local serviceprocessor. In such a system, the local service processor can providepower profiles and analytics of the storage system and individualstorage devices in the system.

FIG. 1 depicts an internal block diagram of a storage device 10according to an embodiment of the present invention. While diagramdepicts features relevant to the illustrated embodiment of theinvention, the storage device 10 may include additional components. Insome embodiments, the storage device 10 may be an SSD, an Ethernet SSD(eSSD), an NVMe SSD, an NVMe-oF SSD, a SAS or SATA SSD.

The storage device 10 includes internal components, including acontroller 11, a memory 12, flash dies 13, a power metering unit (PMU)14 and a connector 15. The controller 11, as known as the processor,implements firmware to retrieve and store data in the memory 12 andflash dies 13 and to communicate with a host computer. In someembodiments, the controller 11 may be an SSD controller, an ASIC SSDcontroller, or an NVMe-oF/EdgeSSD controller. The memory 12 can be arandom access memory such as DRAM or MRAM and the flash dies 13 may beNAND flash memory devices, though the invention is not limited thereto.The controller 11 can be connected to the memory 12 via memory channel22 and can be connected to the flash dies 13 via flash channels 23. Thecontroller 11 can communicate with a host computer via a host interface20 that connects the controller 11 to the host computer through theconnector 15. In some embodiments, the host interface 20 may be a PCIeconnection, an Ethernet connection or other suitable connection. Theconnector 15 may be U.2/M.2 connectors or other suitable connector(s).The PMU 14 allows the storage device 10 to support power managementcapabilities by measuring actual power consumption of the storage device10.

The storage device 10 is supplied power through the connector 15 viapower rails or pins 30. In examples in which the connector 15 is a PCIeconnector, the pins 30 may be 12 V and 30 V pins. In examples in whichthe connector 15 is a U.2 connector, the pins 30 may be 5 V and 12 Vpins (an NVMe SSD may only use the 12 V pin, while a SAS or SATA SSD mayuse both rails). Power rails 30 supply power to the various componentsof the storage device 10. For example, the power rail may supply powerto the various components of the storage device 10 via the PMU 14 andvarious intermediary voltage rails. An embodiment of this is shown inFIG. 1, in which the power rails 30 supply power to the PMU 14, whichthen distributes power to other components of the storage device 10. Forexample, the PMU 14 drives power to the flash dies 13 via flash voltagerails 33. The PMU 14 may similarly drive all power rails to the memory12 via memory voltage rails 32. Power can be supplied to the controller11 by the PMU 14 through multiple voltage rails, such as, for example, acore voltage rail 34, an I/O voltage rail 35 and one or more othervoltage rails 36. Additional voltage rails, such as an additionalvoltage rail 37, may be included to connect other various componentsthat may be included in the storage device 10. The various voltage rails30, 33, 34, 35, 36, 37 used in the storage device 10 can be in a rangeof from 12V down to 0.6V, including 12V and/or 3.3V rails, for example,when the storage device 10 is an NVMe SSD. While, in the embodimentsshown in FIG. 1, the voltage regulators are built into (or integratedwith) the PMU 14, embodiments of the present invention are not limitedthereto, and the voltage regulators may be external to the PMU 14.

In addition to supplying power to the storage device 10, power supplyrails 20 are provided by the PMU 14 inside the storage device 10 togenerate power consumption measurements (“power measurements”) of thevarious voltages rails used by the components of the storage device 10,for example, used by components such as the controller 11, the flashdies 13, the memory 12 and other various components that may be includedin the storage device 10. In some embodiments, the PMU 14 can beprogrammed to support get/set Power State by Power Info from the hostcomputer or BMC.

The PMU 14 can measure the amount of current drawn on various voltagerails it is driving, for example, voltage rails 32, 33, 34, 35, 36 and37. The PMU can output power measurements including the average, minimumand maximum voltage usage by the voltage rails 32, 33, 34, 35, 36 and 37of the storage device 10. In some embodiments, the PMU 14 can meter eachvoltage rail 32, 33, 34, 35, 36 and 37 individually, with the summationof all voltage rails 32, 33, 34, 35, 36 and 37 used by the storagedevice 10 being the total power consumed by the storage device 10. Thepower measurements metered at the PMU 14 can be read by the controller11 using a PMU/controller interface 41. In some embodiments, thePMU/controller interface 41 may be an I2C/SMBus. The controller 11 canthen provide these power measurements to a local service processor 50(see FIG. 3), such as a BMC, via either the host interface 20 or aseparate controller/host interface 42. If a separate controller/hostinterface or side band bus 42 is used, that interface may be anI2C/SMBus. If the controller/host interface 42 is a PCIe connection, thecontroller 11 can provide power measurements to the local serviceprocessor 50 via NVMe-MI or MCTP protocols, as shown in FIG. 4. The PMU14 can report/output the power measurements periodically as specified bythe local service processor 50 or passively keep track via internalcounters which are accessible to the local service processor 50.

FIG. 2 is a flow chart of a method for collecting power consumptionmeasurements from the PMU 14 of the storage device 10. As shown in FIG.2, power measurements can be read at predetermined intervals. Forexample, the power measurements can be read from the PMU 14 of thestorage device 10 at the user's configurable frequency such as 1 second,5 seconds, more than 5 seconds, or every few minutes. In otherembodiments, that storage device 10 can read the power measurements onlyas needed (see, e.g., FIGS. 8 and 9), for example, at the completion ofa specific job. The frequency at which the power measurements are readis hereinafter called a time unit.

For every time unit, the controller 11 prepares (S1) to receive powermeasurements from the PMU 14 for the various voltage rails 30, 33, 34,35, 36, 37. The controller 11 queries (S2) the PMU 14 to determine ifpower measurements from all rails have been completed. If no, then aread request (S3) is sent to a DC-DC regulator at the PMU 14corresponding to a voltage rail for which power measurements have notbeen received (the PMU 14 may include a number of DC-DC regulators eachcorresponding to unique voltage rail). This read request may be send viaan I2C protocol via the PMU/controller interface 41. When the powermeasurement is received from the PMU 14, the power measurement is thenannotated with a timestamp (S4) and a Host ID (S5). The received powermeasurement is then saved (S6) to a power log. The power log may includeinternal register(s) or may be included as part of the PMU's embeddednon-volatile memory.

Once the received power measurement is saved, the PMU 14 is againqueried (S7) until all power measurements are received from the variousvoltage rails 30, 33, 34, 35, 36, 37. Once all power measurements arecomplete and the annotated power measurements are saved in the powerlog, these power measurements persist (S8) in the power log throughresets and power cycles.

In addition to the above annotations, the power log pages can alsoinclude any or all of the following: Namespace ID, NMV Set, read I/Os,write I/Os, SQ ID, Stream ID, and other suitable parameters. Thecontroller 11 also implements actual power (AP) registers which areaccessible by the local service processor 50. This allows a variety ofparameters associated with the storage device and the power measurementsto be mapped with fine granularity.

In some embodiments, the power log can be special proprietary or vendordefined log pages. The power log can be read by the local serviceprocessor 50 using existing standard protocols through either the hostinterface 20 or the separate controller/host interface or side-band bus42, whichever is used. For example, the power log can be read by a BMCusing the NVMe-MI protocol via the controller/host interface 42, whichmay be a SMBus or PCIe.

The above method provides dynamic, real-time output of actual powerconsumption measurements without affecting the I/O of the storagedevice. With the power measurement information, the local serviceprocessor can implement power budgets and allocate power to the storagedevice based on its actual power usage. For example, the local serviceprocessor can implement power budgets similar to existing industrystandards for allocated power budget registers. Also, the storage devicecan report real time power consumption to system management software,such as Samsung's DCP or Redfish.

FIG. 3 is a block diagram of a storage system 100 incorporating multiplestorage devices 10. The storage system 100 includes the local serviceprocessor 50 attached to multiple storage devices 10. Each storagedevice 10 has a PMU 14 to measure power consumption as described abovewith respect to FIGS. 1 and 2. In the illustrated embodiment, thestorage devices 10 provide power measurements to the local serviceprocessor 50 via the controller/host interface 42. In some embodiments,the controller/host interface 42 may be an I2C/SMBus or PCIe bus. Thepower measurements may be transferred to the local service processor 50using NVMe protocols, such as NVMe-MI, MCTP over PCI-e, or I2C Busprotocols. If the storage device 10 is connected via a SMBus/I2Cconnection, the local service processor 50 can even access the power logduring a power failure using these existing standard protocols.

FIG. 4 is a block diagram of an embodiment of the storage system 100 ofFIG. 3 in which a PCIe switch 60 is used. In this embodiment, thestorage devices 10 are connected to the local processor 50 via the PCIeswitch 60. The power measurements may be transferred to the localservice processor 50 via the PCIe switch 60 using suitable protocolssuch as, for example, NVMe-MI and/or MCTP.

In the embodiments of FIGS. 3 and 4, the local service processor 50 andthe multiple storage devices 10 can be housed within the same chassisallowing the local service processor 50 to process the powermeasurements of the multiple storage devices 10 according to chassispower management requirements; however, the invention is not limitedthereto. For example, power measurements can also be processed at theindividual storage device level.

In embodiments in which the power measurements are transferred to thelocal service processor 50 using NVMe protocols, NVMe specifications candefine power measurements and their process mechanism. Based on thismechanism, the storage devices 10 (e.g., an NVMe SSD) can support powermanagement either queried by the local service processor 50 (FIG. 5) orset by the local service processor 50 (FIG. 6).

FIG. 5 is a diagram depicting an embodiment in which power measurementsare transferred to the local service processor 50 based on a query fromthe local service processor 50. In this embodiment, the local serviceprocessor 50 queries the power measurement information by sending aGetFeature command (S10), for example, FeatureID=0x2, to the firmware ofthe controller 11 for the storage device 10 from which the local serviceprocessor 50 is seeking power measurement information. The controller'sfirmware then fetches (S11) the power measurement information from thePMU 14. The firmware of the controller 11 receives the information andsends (S12) that information via direct memory access (DMA) to the localservice processor 50. The controller's firmware then sends (S13) acompletion notice to the local service processor 50 to signal completionof the query. This embodiment allows for real-time retrieval of powermeasurements from the storage device 10.

FIG. 6 is a diagram depicting an embodiment in which power measurementsare set by the local service processor 50. In this embodiment, the localservice processor 50 sets the power measurement information(hereinafter, called the power measurement budget) by sending aSetFeature command (S20), for example, FeatureID=0x2, to the firmware ofthe controller 11 for the storage device 10 for which the local serviceprocessor 50 intends to set the power measurement budget. Thecontroller's firmware then uses DMA to request (S21) the powermeasurement budget from the local service processor 50. The firmware ofthe controller 11 receives the information and sets (S22) the powermeasurement budget of the PMU 14. In response, the controller's firmwareprocesses the new power state transaction. In order to process the newpower transaction, the controller's firmware queries the current powerstate job in the PMU 14 to ensure that all tasks that rely on thecurrent power state are fully completed successfully. Then, the firmwarechanges the current power state from the current one to the next onerequired by the power measurement budget. The controller's firmwarestarts to process new tasks which rely on the power state using theallocated power measurement budget. The controller's firmware then sends(S23) a completion notice to the local service processor 50 to signalthat the new power state has been set.

By enabling this SetFeature function, the local service processor 50 cancontrol and throttle the power consumption of a particular storagedevice 10 to meet an allocated power budget of the local serviceprocessor 50. The controller 11 can enforce the power budget allocationsprogrammed by the local service processor 50. If the actual powerconsumption exceeds the set threshold, the controller 11 can throttlethe I/O performance for that parameter in order to minimize powerconsumption and to stay within the allocated power budget. Thecontroller 11 can, for example, self-adjust by lowering the internalpower state automatically when exceeding the allocated power budget. Thecontroller 11 can then report back to the local service processor 50 sothat the local service processor 50 can reallocate the available powerto some other devices which may need additional power. The controller 11may also collect statistics about such performance throttling on a finegranularity.

FIG. 7 shows an example of a power policy which can be used by the localservice processor 50 to control power consumption of a storage device10. The local service processor 50 can manage the power policy bymonitoring each storage device 10 in the storage system and instructingeach storage device 10 to maintain its respective allocated powerbudget. For example, if a storage device 10 changes from operating atnormal 61 to operating at greater than 90% of its allocated powerbudget, as shown at 62, the controller 11 may throttle I/O performanceby, for example, introducing additional latency of a small percentage(e.g., 10% or 20% of idle or overhead). However, if the current state isgreater than 100% of its allocated power budget, as shown at 63, thecontroller 11 may introduce a much bigger latency (e.g., 50% or larger)or may introduce delays to NAND cycles, etc., in order to throttle thestorage device 10 to meet its allocated budget. If the storage device 10continues to exceed its allocated budget despite the introducedlatencies, the local service processor 50 may execute shutdowninstructions 64 to shutdown the device 10 or the controller 11 mayshutdown itself.

In further embodiments, the local service processor 50 can also monitorand detect thermal load increases (temperature rises) or operate theresource during peak utility rate such as hot day times or duringbrown-out periods to ensure that each storage device 10 is behaving asintended performance-wise.

The above feature makes the storage device capable of autonomousoptimizing power vs. performance vs. assigned power budget/state.

FIG. 8 is a diagram depicting a further embodiment in which powermeasurements are stored in the controller memory buffer until fetched bythe local service processor 50. In this embodiment, the controller 11can store the power measurements locally in its own memory 12 untilrequested by the local service processor 50. For example, the controller11 could store the power measurement information in a controller memorybuffer of the memory 12 in an embodiment in which the storage device 10is an NVMe SSD. The NVMe specification define the controller memorybuffer (CMB), which is a portion of the storage device's memory, but isassigned by the host/local service processor and owned by the host/localservice processor logically.

The firmware of the controller 11 can fetch power measurementinformation from the PMU 14 and store it in the control memory buffer ofthe memory 12. The control memory buffer can be updated at anydesignated time unit. The local service processor 50 can then query thepower measurement information by reading the power measurements directlyfrom the controller memory buffer of the memory 12. The powermeasurements can be read from the control memory buffer via thecontroller/host interface 42. If the controller/host interface 42 isPCIe, the power measurement information can go through the PCIe todirectly process memRd/memWr based on the

BAR configuration in order to read from the control memory buffer. Inother embodiments, the power measurement information can go through sideband such as SMBus or I2C to directly access the control memory buffer.

Alternative to FIG. 8, the storage device 10 can be configured so thatthe PMU 14 is directly accessible by the local service processor 50 inorder for the local service processor to be able to access the powermeasurement information when desired/needed and in real-time.

FIG. 9 is a diagram depicting an embodiment in which power measurementstaken by the PMU 14 are directly accessible to the local serviceprocessor 50. In this embodiment, the storage device 10 can beconfigured with an assistant bus, such as, for example, I2C or AXI, toallow direct access to the PMU 14 by the local service processor 50.This allows the local service processor 50 to be able to process thepower measurement information by accessing the PMU 14 directly andallows for retrieval of power measurements in real-time.

FIG. 10 is an example of a power log 70 according to an embodiment ofthe present invention. As illustrated in this embodiment, a storagedevice 10 may have, for example, up to 32 Power States (PowerState) 71,which are recorded in the power log 70. Each PowerState 71 haspredefined performance information, a Maximum Power (MP) 72 capable ofbeing utilized in that Power State 71 and an Actual Power (AP) 73actually being used at that PowerState. AP 73 is a measured periodaccording to the time unit (e.g., 1 minute) and Workload/QoS. In thecurrent embodiment, each row in the power log 70 represents a powerstate which has been defined in the NVMe Specifications 1.3. Forexample, there are total 32 Power State defined in NVMe Specifications.In some embodiments, a vendor-specific definition can be used for eachPowerState 71.

The power log 70 can include in its table entries the variousPowerStates 71 and each PowerState's respective MP 72, AP 73 andadditional information for identifying the power measurements and arelationship among Max Power/Power State, Actual Power, and QoS. QoSinformation can include, for example, current Entry Latency (ENTLAT),current Exit Latency (EXTLAT), RRT (Relative Read Throughput), RWT(Relative Write Throughput) and other suitable variables.

FIG. 10 illustrates a Power State_3 with a defined Max Power=20 W.However, the storage device 10 at this Power State currently consumes anActual Power=19 W. Current QoS is shown in other columns such as RRT=2,RWT=2, ENTLAT=20 us and EXTLAT =30 us. If applications 80 run on thestorage system 200 expect the best QoS (such as the best RRT & RWT),those applications 80 could instruct the local service processor 50 togive more power to the storage device 10 by transferring from PowerState_3 to Power State_0.

The current PowerState 71 is retrieved by the local service processor 50through the GetFeature (FeatureID=0x2), as discussed with respect toFIG. 5. An expected power state (i.e. power measurement budget) can beset by the local service processor 50 through the SetFeature(FeatureID=0x2), as discussed with respect to FIG. 6. Otherpower-related information can be managed by local service processor 50through VUCmd (Vendor Unique Cmd) or directly accessed through the localservice processor 50. For example, if the user would like to get powermeasurement information which is not defined in the NVMe specification,a VUCmd can be used to allow host retrieve such non-standard powerinformation, similar to LogPage.

FIG. 11 is an illustrative method of how a storage system 200 managesthe power reporting of multiple storage devices 10 in its chassis.According to this method, each PMU 14 of each storage device 10 measuresthe current AP 73 and stores the information in the power log 70, whichis queried and/or retrieved (S50) by the local service processor 50. Thelocal service processor 50 then updates/uploads (S51) the power log 70from the local service processor 50 to the storage system 200. Variousapplications 80 in the storage system 200 can analyze (S52) the powerlogs 70 of the storage devices 10 in the chassis at the local serviceprocessor 50. The results of these analyses can determine how toallocate power for better performance, e.g., whether more power needs tobe allocated to a particular PowerState 71 or whether power should bereallocated from one PowerState 71 to another to meet QoS demands. Forexample, the local service processor 50 can request (S53) that thestorage device 10, as illustrated with respect to the center storagedevice 10 shown in FIG. 10, transfer Max Power State, in this example,from PowerState 3 to PowerState 0. The local service processor 50 canthen either assign a new MP 72 to the storage devices 10 or can request(S54) a power distribution unit (PDU) 90 to assign a new MP 72 budget tothe storage devices 10, i.e. redistributing power allocations. If thePDU 90 is used, the PDU will then assign (S55) the new MP 72 to thestorage devices 10. The PDU 90 may be an independent component locatedin the chassis and may responsible for distributing MP to each storagedevice 10. The local service processor 50 then updates (S56) the powerlog 70 with the changes.

As discussed above, once the local service processor 50 has access andcan read the power measurements, the local service processor 50 can thenuse that information to create graphs or histograms to trend projectionsand to run diagnostics.

Embodiments of the present invention also enable the local serviceprocessor to provide individual actual power profiles of each storagedevices in the system to software developers, cloud service providers,users and others by allowing them to know the actual power consumptionof their workloads consumed on each storage device. This provides theability for software developers/users to optimize performance based onthe actual cost of energy and also allows cloud service providers toprovide more accurate billing of storage system users based on actualpower consumption. Embodiments of the present invention can also providebetter policing and tracking of storage devices violating an allocatedpower budget.

Embodiments of the present invention may be used in a variety of areas.For example, the embodiments of the present invention provide buildingblocks of crucial information that may be used for analysis purposes forartificial intelligence software, such as Samsung's DCP. The embodimentsalso provide information that may be useful to an ADRC (ActiveDisturbance Rejection) High Efficient Thermal control based system.

Although exemplary embodiments of the present invention have beendescribed, it is understood that the present invention should not belimited to these exemplary embodiments but various changes andmodifications can be made by one ordinary skilled in the art within thespirit and scope of the present invention as hereinafter claimed byappended claims and equivalents thereof.

FIG. 12 is a block diagram illustrating a storage system 300 utilizing astorage bank 302 and a power distribution unit (PDU) 90, according tosome exemplary embodiments of the present invention.

In some embodiments, the storage bank (e.g., an Ethernet SSD chassis orJust-a-bunch-of-flashes (JBOF)) 302 includes a plurality of storagedevices 10, and the PDU 90 includes a plurality of power supply units(PSUs or power supplies) 304 for supplying power to the storage devices10 of the storage bank 302 under the direction of the local serviceprocessor (or BMC) 50. In some embodiments, the PSUs 304 areinterchangeable, that is, each may have the same form factor and thesame power supply capacity (e.g., have same output wattage); however,embodiments of the present invention are not limited thereto, and one ormore of the PSUs 304 may have a power supply capacity that is differentfrom other PSUs 304. In some examples, the plurality of PSUs 304 may bein an N+1 configuration in which N (an integer greater than or equal to1) PSUs are sufficient to service the power needs of the storage bank302, and an additional PSU 304 is provided as redundancy, which may beactivated in the event that any of the PSUs experiences a failure.

As shown in FIG. 12, in some embodiments, the PSUs 304 may be coupledtogether using a switch network (e.g., a FET network) 305, rather thandirectly connected to the power bus 306, in order to protect the powerbus 306 from electrical short circuits and transients when other PSUs304 are connected. The switch network may include a plurality ofswitches (e.g., transistors) that are connected to the plurality of PSUs304, on one end, and connected to the power bus 306, at the other end.According to some embodiments, the switches are independently controlledby the local service provider (BMC) 50, so that ny one of the PSUs 304may be connected to, or disconnected from, the power bus 306, based on acontrol signal from the local service provider 50.

According to some embodiments, each storage devices 10 is configured toreport its actual power consumption to the local service processor 50via, for example, SMBus or PCI-e, and by, for example, NVMe-MI or MCTPprotocols. The actual power consumption is measured by the PMU (i.e.,power meter) 14, which may be internal to (e.g., integrated within) thestorage device 10 (as shown in FIG. 12) or be external to, but coupledto, the storage device 10. The power consumption reporting enables thelocal service processor 50 to provide power profiles and performanalytics on the storage bank 302, which can in turn be used fordiagnostics as well as offering value added services. This also allowseach storage device 10 to more flexibly manage its own power usage asdictated by the system administrator 308, via the local serviceprocessor 50.

FIGS. 13A-13D illustrate histograms of power consumption of a storagesystem as generated by the local service processor 50, according to someexemplary embodiments of the present invention.

According to some embodiments, the local service processor 50 reads thepower measurements periodically from the storage devices 10. In sodoing, local service processor 50 may use NVMe-MI protocol over SMBus orPCIe to read the power log 70 pages, according to some examples. Thelocal service processor 50 may then process the read power data togenerate power usage trends, such as whole power usage of the storagebank 302 over time (e.g., per hour, during day time, night time,weekdays, or weekends, etc.), each storage device's 10 power consumptionover time, relative power consumption of the storage devices 10 in astorage bank 302, and/or the like. In addition, the local serviceprocessor 50 may generate many derivative/additional graphs to learnabout the power consumption behavior with respect to time, user,activity, etc. The local service processor 50 may also utilize such datafor diagnostics purposes, power provisioning, future needs, cooling, andplanning, etc.

As an example, FIG. 13A illustrates the power consumption of a singlestorage device 10 over time. In FIG. 13A, the Y axis represents powerconsumption in terms of Watts, and the X axis represents time in termsof hours.

In some embodiments, the local service processor 50 manages host accesspolicies, and receives raw power data and host IDs of active storagedevices. Thus, according to some embodiments, the local serviceprocessor 50 is cognizant/aware of which host or application isaccessing each storage device 10 at any given time, and is able tocombine this information with power usage metrics to profile the powerconsumption by various hosts or applications. Such information canprovide deeper insights into storage power needs to various applicationsand can be used to calculate the storage costs per host or applicationmore accurately.

As an example, FIG. 13B illustrates power consumption by different hostsor applications. In FIG. 13B, the Y axis represents average powerconsumption in Watts over a period of time (e.g., per hour, day, etc.),and the X axis represents the host ID or application ID.

According to some embodiments, the local service processor 50 is capableof using power usage metrics for diagnostic purposes. In someembodiments, when abnormal power consumption is observed for a storagedevice 10, the local service processor 50 may alert the storageadministrator 308. The abnormal power consumption may be a result of afault within the storage device 10, or may be due to anomalous activityof the host or application that is accessing the storage device 10. Forexample, the faults may be a result of flash die or flash channelfailures, which may initiate RAID like recovery mechanism consumingexcess power; or higher bit rate errors in the media or volatile memory,which may cause error correction algorithms not to converge and spendmore time and energy on a process. The local service processor 50 mayquery storage device health and status logs, such as SMART Logs, as wellas proprietary diagnostic logs to assess abnormal behavior. Based on thepolicies set by the administrator 308, some of the abnormal behavior maybe alerted to the administrator 308 for further action.

For example, FIG. 13C illustrates a potential fault detected in astorage device 10 when the power consumption per hour suddenly spikesabout normal levels (e.g., 3-10 W/hr) to close to maximum values (e.g.,around 25 W). In FIG. 13C, the Y axis represents average powerconsumption in Watts, and the X axis represents time in terms of hours.Thus, in some embodiments, the criterion for fault detection may be thederivative of power consumption being greater than a set threshold.However, embodiments of the present invention are not limited thereto,and the actual power consumption may be measured against storage deviceperformance to determine if a fault has occurred or not. In someexamples, the fault detection criteria/policy may be set by theadministrator 308.

Further, FIG. 13D illustrates an example, in which a potential fault isdetected in a storage device 10 (e.g., the storage device in slot #8).In this example, the storage device 10 may be expected to consume amaximum power of about 25 W at 1 MIOPS (one million input/outputoperations per second) of performance. However, if the average powerconsumption of storage device in slot #8 reaches the maximum power ofabout 25 W, but the average performance is much lower than 1 MIOPs, thenthe local service processor may tag the storage device in slot #8 aspotentially faulty or at least a good candidate for further faultanalysis.

Accordingly, aspects of the present invention provide the building blockof crucial information for other artificial intelligence SW to analyze.In addition, it also provides useful information for an ADRC (activedisturbance rejection control), high-efficiency, thermal-control basedsystem to take advantage of.

FIG. 14 is flow diagram illustrating the process 400 of managingoperations of the PDU 90, according to some exemplary embodiments of thepresent invention.

According to some embodiments, the local service provider 50 manages(e.g., optimizes) operations of the PDU 90 by dynamically monitoring theoperation of the PSUs 304 of the PDU 90 and ensuring that active PSUs304 operate in their high power-efficiency range. In so doing, the localservice provider 50 determines (S100) whether the PDU 90 includesmultiple active PSUs 304 or not. The active PSUs 304 may be connected tothe power bus 306 through the switch network (i.e., have thecorresponding witches turned on), and the deactivated PSUs 304 may bedisconnected from the power bus 306 (e.g., by having the correspondingswitches turned off). In some embodiments, the local service provider 50determines the status of each PSU 304 in the PDU 90 through a bus (e.g.,SMBus/PMBus), and is thus able to determine the number of PSUs 304 atthe PDU 90. In some examples, the local service provider 50 reads thePSU status register of each PSU 304 present in the PDU 90 to determineits status (i.e., active/enabled or deactivated/disabled). If only oneactive PSU 304 is present, the local service provider 50 proceed todetermine (S114) if the active PSU 304 is the only one PSU 304 presentand is in HA mode (more on this below). Otherwise, the local serviceprovider 50 determines (S102) whether the total power consumption of thestorage bank 302 is less than a first percentage threshold (e.g., 40% ora value between 30% to 50%) of the load of each of the active PSUs 304.In some embodiments, the local power processor 50 does so by obtainingthe actual power consumption of each storage device 10, as measured bythe corresponding PMU 14, and adding together the actual powerconsumptions. In some examples, the local service provider 50 may obtainthe actual power consumption of each storage device 10 byquerying/retrieving the power log 70 from the storage device 10 or thePMU 14 corresponding to the storage device 10 (which may be internal toor external to the storage device 10).

If the total power consumption is less than the first percentagethreshold of the load of each of the active PSUs 304, the active PSUs304 may be operating in low power efficiency mode, which may beundesirable. As such, the local service provider 50 disables an activePSU 304 (S104), waits (S106) for a period of time (e.g., seconds orminutes), and rechecks (S102) whether the total power consumption of thestorage bank 302 is still less than the first percentage threshold ofthe load of each of the active PSUs 304. If so, the loop continues andthe local service provider 50 continues to disable the active PSUs 304one by one until the total power consumption is equal to or greater thanthe first percentage threshold of the load of each of the active PSUs304.

At that point, the local service provider 50 proceeds to determine(S108) whether the total power consumption of the storage bank 302 isgreater than a second percentage threshold (e.g., about 90% or a valuebetween 85% and 95%) of the load of each of the active PSUs 304. If so,the active PSUs 304 may be operating in high-power state, which may bedetrimental to the longevity of the PSUs 304 if prolonged. As such, thelocal service provider 50 enables (i.e., activates) a disabled (i.e., adeactivated) PSU 304 (S110), waits (S112) for a period of time (e.g.,seconds or minutes), and rechecks (S108) whether the total powerconsumption of the storage bank 302 is still equal to or greater thanthe second percentage threshold of the load of each of the active PSUs304. If so, the loop continues and the local service provider 50continues to enable the active PSUs 304 one by one until the total powerconsumption is less than the second percentage threshold of the load ofeach of the active PSUs 304.

At that point, the local service provider 50 proceeds to determine(S114) if only one PSU 304 is present in the PDU 90 while the storagesystem 300 is in high availability (HA) mode, which indicates multi-path10 mode and N+1 redundant PSUs. Generally, in HA mode, the storagesystem 300 is in multi-path 10 mode and N+1 redundant PSUs are presentto ensure that there is no single point of failure. As such, when onlyone PSU 304 is present in the PDU 90 while the system 300 is in HA mode,the local service provider 50 issues a warning (e.g., a criticalwarning) message (S116) to the system administrator 308 to installanother redundant PSU 304 in the PDU 90. Otherwise, the system isoperating normally and no warning message is sent to the systemadministrator 308.

FIG. 15 is flow diagram illustrating a process 500 of managing thestorage devices 10 of the storage system 300, according to someexemplary embodiments of the present invention.

According to some embodiments, the local service provider 50 manages(e.g., optimizes) storage devices 10 by dynamically adjusting (e.g.,lowering) their maximum power range or power cap based on the currentworkload of the storage bank 302.

In some embodiments, the local service provider 50 identifies (S118)which storage devices 10 of the storage bank 302 are in an idle state orconsume near-idle power. Herein, an idle state may refer to anoperational state in which a storage device 10 does not have any activeor outstanding host commands such as read or write in its command queuefor a period of time. That is to say that the host command queues of thestorage device controller have been empty for a period of time, whichmay be programmable (e.g., by the system administrator 308). Near-idlepower may be any power consumption that is below a set threshold, whichmay be programmable (e.g., by the system administrator 308). In someembodiments, the local power processor 50 obtains the actual powerconsumption of each storage device 10, which is measured by thecorresponding PMU 14, by querying/retrieving the power log 70 from thestorage device 10. The local service provider 50 then compares theactual power consumption with an idle power level. If consumed power ofthe storage device 10 is at or below the idle power level, the storagedevice is identified as being in an idle state. The local serviceprovider 50 then instructs (S120) the identified storage devices 10 tooperate at a lower power cap. For example, the local service processor50 may instruct each of the identified storage devices 10 to changepower states to a power state having a lower maximum power rating (e.g.,change from PowerState 2 to PowerState 5). This may be done based on apower policy that is implemented by the local service provider 50 (andis, e.g., defined by the system administrator 308), which associateseach power state to a range of actual power consumption.

According to some embodiments, the local service provider 50 identifies(S122) which storage devices 10 consume power at a level less than athreshold power level. In some examples, the threshold may be set at 75%of maximum power, which may be 25 W, or 75 W, etc., depending on thekind of PSUs and/or power connectors used.

In some embodiments, the local power processor 50 obtains the actualpower consumption of each storage device 10, which is measured by thecorresponding PMU 14, by querying/retrieving the power log 70 from thestorage device 10. The local service provider 50 then compares theactual power consumption with threshold power level to determine ifconsumed power of the storage device 10 is below the threshold powerlevel. The local service provider 50 then dynamically instructs theidentified storage devices 10 to operate at a power cap corresponding tothe first level (e.g., at 75% or 80% of maximum power), as opposed tothe default power cap of 100% maximum power. Because the powerefficiency of a PSU drops as it reaches its maximum load capacity,lowering the power cap of the storage devices 10 may bring down theoverall power usage of the storage bank 302, thus allowing the PSU tooperate at a lower power level and at a higher (e.g., peak) powerefficiency range. This may be particularly desirable in large datacenters, where overall power usage and cooling is a great concern.

In some examples, the local service provider 50 may dynamically instructeach of the identified storage devices 10 to operate at a lower powercap by instructing them to change their power state to one where themaximum power corresponds to (e.g., is at or less than) the thresholdpower level (e.g., the power states may be changed from PowerState 0 toPowerState 1).

In some embodiments, the local service provider 50 identifies (S126)which storage device slots are empty (i.e., not occupied by, orconnected to, any storage device 10). In some examples, each storagedevice 10 may have a presence pin on the slot connector 15, which isused by the service provider 50 to determine whether the slot is emptyor occupied by a storage device 10. If any of the empty slots havecorresponding PMU 14 that are external to (i.e., not integrated with,and outside of) their corresponding storage devices 10 (e.g., may be ata power distribution board or at a mid-plane of the storage chassis),the local service provider 50 instructs (S128) that these PMUs 14operate at lower power caps (e.g., operate at the lowest power state,PowerState 31) or disable/deactivate altogether. This will allow thestorage bank 302 to eliminate or reduce unnecessary power usage.

While operations S118-S120, S122-S124, and S126-S128 are ordered in aparticular sequence in FIG. 15, embodiments of the present invention arenot limited thereto. For example, the operations S118-S120 can beperformed after either or both of operations S122-S124 and S126-S128,and operations S126-S128 may be performed before either or both ofoperations S118-S120 and S122-S124.

The operations performed by the local service provider 50 (e.g.,processes 400 and 500) may be described in terms of a software routineexecuted by one or more processors in the local service provider 50based on computer program instructions stored in memory. A person ofskill in the art should recognize, however, that the routine may beexecuted via hardware, firmware (e.g. via an ASIC), or in combination ofsoftware, firmware, and/or hardware. Furthermore, the sequence of stepsof the process is not fixed, but may be altered into any desiredsequence as recognized by a person of skill in the art.

1-9. (canceled)
 10. A storage system comprising: a plurality of storagedevices comprising a first storage device and a second storage deviceconfigured to measure a power consumption of the first storage deviceand a power consumption of the second storage device; a processor incommunication with the plurality of storage devices; and a memory havingstored thereon instructions that, when executed by the processor, causethe processor to perform: determining whether the first storage deviceis in a first state; based at least in part on determining that thefirst storage device is in the first state, instructing the firststorage device to operate at a lower power cap; determining whether thesecond storage device is consuming power under a threshold power level;and based at least in part on determining that the second storage deviceis consuming power under the threshold power level, instructing thesecond storage device to operate at or below the threshold power level.11. The storage system of claim 10, wherein the first state is an idlestate, and wherein the determining whether the first storage device isin the first state comprises: obtaining the power consumption of thefirst storage device by retrieving a power log from the first storagedevice; comparing the power consumption of the first storage device withan idle power level; and determining whether the first storage devicehas a power consumption that is at or below the idle power level. 12.The storage system of claim 11, wherein the power log is configured tostore an actual power consumption of a corresponding storage device asmeasured by a corresponding power meter.
 13. The storage system of claim12, wherein the corresponding power meter is internal to the firststorage device.
 14. The storage system of claim 12, wherein thecorresponding power meter is external to and coupled to the firststorage device.
 15. The storage system of claim 11, wherein the powerlog is configured to record power measurement information measured bypower meter corresponding to the first storage device.
 16. The storagesystem of claim 10, wherein the instructing the first storage device tooperate at the lower power cap comprises: instructing the first storagedevice to change to a power state having a lower maximum power rating.17. The storage system of claim 10, wherein the determining whether thesecond storage device is consuming power under the threshold power levelcomprises: obtaining the power consumption of the second storage deviceby retrieving a corresponding power log from the second storage device;comparing the power consumption of the second storage device with thethreshold power level; and determining whether the second storage devicehas a power consumption that is below the threshold power level.
 18. Thestorage system of claim 10, wherein the instructing the second storagedevice to operate at or below the threshold power level comprises:instructing the second storage device to change to a power state havinga maximum power rating corresponding to the threshold power level. 19.The storage system of claim 10, wherein the instructions further causethe processor to perform: determining whether one or more storage slotsare not occupied by any storage device; and based at least in part ondetermining that the one or more storage slots are not occupied by anystorage device: identifying one or more power meters associated with theone or more storage slots; and instructing the one or more power metersto operate at a lower power capacity.
 20. The storage system of claim19, wherein the instructing the one or more power meters to operate atthe lower power capacity comprises: instructing the one or more powermeters to operate at a lowest power state.
 21. The storage system ofclaim 19, wherein the instructing the one or more power meters tooperate at the lower power capacity comprises: instructing the one ormore power meters to deactivate.
 22. A method of managing a storagesystem comprising a first storage device and a second storage device,the method comprising: determining whether the first storage device isin a first state; based at least in part on determining that the firststorage device is in the first state, instructing the first storagedevice to operate at a lower power cap; determining whether the secondstorage device is consuming power under a threshold power level; andbased at least in part on determining that the second storage device isconsuming power under the threshold power level, instructing the secondstorage device to operate at or below the threshold power level.
 23. Themethod of claim 22, wherein the first state is an idle state, andwherein the determining whether the first storage device is in the firststate comprises: obtaining a power consumption of the first storagedevice by retrieving a power log from the first storage device;comparing the power consumption of the first storage device with an idlepower level; and determining whether the first storage device has apower consumption that is at or below the idle power level.
 24. Themethod of claim 22, wherein the instructing the first storage device tooperate at the lower power cap comprises: instructing the first storagedevice to change to a power state having a lower maximum power rating.25. The method of claim 22, wherein the determining whether the secondstorage device is consuming power under the threshold power levelcomprises: obtaining a power consumption of the second storage device byretrieving a corresponding power log from the second storage device;comparing the power consumption of the second storage device with thethreshold power level; and determining whether the second storage devicehas a power consumption that is below the threshold power level.
 26. Themethod of claim 22, wherein the instructing the second storage device tooperate at or below the threshold power level comprises: instructing thesecond storage device to change to a power state having a maximum powerrating corresponding to the threshold power level.
 27. The method ofclaim 22, further comprising: determining whether one or more storageslots are not occupied by any storage device; and based at least in parton determining that the one or more storage slots are not occupied byany storage device: identifying one or more power meters associated withthe one or more storage slots; and instructing the one or more powermeters to operate at a lower power capacity.
 28. The method of claim 27,wherein the instructing the one or more power meters to operate at thelower power capacity comprises: instructing the one or more power metersto operate at a lowest power state.
 29. The method of claim 27, whereinthe instructing the one or more power meters to operate at the lowerpower capacity comprises: instructing the one or more power meters todeactivate.